From policies to influences: a framework for nonlocal abstraction in transition-dependent Dec-POMDP agents

Decentralized Partially-Observable Markov Decision Processes (Dec-POMDPs) are powerful theoretical models for deriving optimal coordination policies of agent teams in environments with uncertainty. Unfortunately, their general NEXP solution complexity [3] presents significant challenges when applying them to real-world problems, particularly those involving teams of more than two agents. Inevitably, the policy space becomes intractably large as agents coordinate joint decisions that are based on dissimilar beliefs about an uncertain world state and that involve performing actions with stochastic effects. Our work directly confronts the policy space explosion with the intuition that instead of coordinating all policy decisions, agents need only coordinate abstractions of their policies that constitute the essential influences that they exert on each other. As a running example, consider the problem shown in Figure 1, involving two interacting rover agents (among a team of several others) that are exploring the surface of Mars. As shown, the agents perform various tasks (constrained to take place within a window of execution) with nondeterministic duration (D) and quality (Q) outcomes, and in performing their tasks may alter the outcomes of other agents’ tasks. Here, agent 1 may choose to visit and prepare research site C, which will (in expectation) make agent 2’s analysis of site C quicker and more valuable. This problem can be expressed Cite as: From Policies to Influences: A Framework for Nonlocal Abstraction in Transition-Dependent Dec-POMDP Agents (Extended Abstract), S. Witwicki and E. Durfee, Proc. of 9th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2010), van der Hoek, Kaminka, Lespérance, Luck and Sen (eds.), May, 10–14, 2010, Toronto, Canada, pp. Copyright c © 2010, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved. Analyze C outcome: